4 research outputs found
COPAL-ID: Indonesian Language Reasoning with Local Culture and Nuances
We present publicly available COPAL-ID, a novel Indonesian language common
sense reasoning dataset. Unlike the previous Indonesian COPA dataset
(XCOPA-ID), COPAL-ID incorporates Indonesian local and cultural nuances, and
therefore, provides a more natural portrayal of day-to-day causal reasoning
within the Indonesian cultural sphere. Professionally written by natives from
scratch, COPAL-ID is more fluent and free from awkward phrases, unlike the
translated XCOPA-ID. In addition, we present COPAL-ID in both standard
Indonesian and in Jakartan Indonesian--a dialect commonly used in daily
conversation. COPAL-ID poses a greater challenge for existing open-sourced and
closed state-of-the-art multilingual language models, yet is trivially easy for
humans. Our findings suggest that even the current best open-source,
multilingual model struggles to perform well, achieving 65.47% accuracy on
COPAL-ID, significantly lower than on the culturally-devoid XCOPA-ID (79.40%).
Despite GPT-4's impressive score, it suffers the same performance degradation
compared to its XCOPA-ID score, and it still falls short of human performance.
This shows that these language models are still way behind in comprehending the
local nuances of Indonesian.Comment: 8 page
NusaCrowd: Open Source Initiative for Indonesian NLP Resources
We present NusaCrowd, a collaborative initiative to collect and unify
existing resources for Indonesian languages, including opening access to
previously non-public resources. Through this initiative, we have brought
together 137 datasets and 118 standardized data loaders. The quality of the
datasets has been assessed manually and automatically, and their value is
demonstrated through multiple experiments. NusaCrowd's data collection enables
the creation of the first zero-shot benchmarks for natural language
understanding and generation in Indonesian and the local languages of
Indonesia. Furthermore, NusaCrowd brings the creation of the first multilingual
automatic speech recognition benchmark in Indonesian and the local languages of
Indonesia. Our work strives to advance natural language processing (NLP)
research for languages that are under-represented despite being widely spoken